Statistical Learning : stability is sufficient for generalization and necessary and sufficient for consistency of Empirical Risk Minimization
نویسندگان
چکیده
Solutions of learning problems by Empirical Risk Minimization (ERM) – and almost-ERM when the minimizer does not exist – need to be consistent, so that they may be predictive. They also need to be well-posed in the sense of being stable, so that they might be used robustly. We propose a statistical form of leave-one-out stability, called CVEEEloo stability. Our main new results are two. We prove that for bounded loss classes CVEEEloo stability is (a) sufficient for generalization, that is convergence in probability of the empirical error to the expected error, for any algorithm satisfying it and, (b) necessary and sufficient for generalization and consistency of ERM. Thus CVEEEloo stability is a weak form of stability that represents a sufficient condition for generalization for general learning algorithms while subsuming the classical conditions for consistency of ERM. We discuss alternative forms of stability. In particular, we conclude that for ERM a certain form of well-posedness is equivalent to consistency. This report describes research done within the Center for Biological & Computational Learning in the Department of Brain & Cognitive Sciences and in the Artificial Intelligence Laboratory at the Massachusetts Institute of Technology. This research was sponsored by grants from: Office of Naval Research (DARPA) under contract No. N00014-00-1-0907, National Science Foundation (ITR) under contract No. IIS-0085836, National Science Foundation (KDI) under contract No. DMS-9872936, and National Science Foundation under contract No. IIS-9800032 Additional support was provided by: Central Research Institute of Electric Power Industry, Center for e-Business (MIT), Eastman Kodak Company, DaimlerChrysler AG, Compaq, Honda R&D Co., Ltd., Komatsu Ltd., Merrill-Lynch, NEC Fund, Nippon Telegraph & Telephone, Siemens Corporate Research, Inc., The Whitaker Foundation, and the SLOAN Foundations.
منابع مشابه
Statistical Learning : CVloo stability is sufficient for generalization and necessary and sufficient for consistency of Empirical Risk Minimization
Solutions of learning problems by Empirical Risk Minimization (ERM) – and almost-ERM when the minimizer does not exist – need to be consistent, so that they may be predictive. They also need to be well-posed in the sense of being stable, so that they might be used robustly. We propose a statistical form of stability, defined in terms of the property of cross-validation leaveone-out (CVloo) stab...
متن کاملStatistical Learning : CVEEEloo stability is sufficient for generalization and necessary and sufficient for consistency of Empirical Risk Minimization
Solutions of learning problems by Empirical Risk Minimization (ERM) – and almost-ERM when the minimizer does not exist – need to be consistent, so that they may be predictive. They also need to be well-posed in the sense of being stable, so that they might be used robustly. We propose a statistical form of leave-one-out stability, called CVEEEloo stability. We prove that for bounded loss classe...
متن کاملStatistical Learning : LOO stability is sufficient for generalization and necessary and sufficient for consistency of Empirical Risk Minimization
Solutions of learning problems by Empirical Risk Minimization (ERM) – and almost-ERM when the minimizer does not exist – need to be consistent, so that they may be predictive. They also need to be well-posed in the sense of being stable, so that they might be used robustly. We propose a statistical form of stability, defined as leave-one-out (LOO) stability. We prove that for bounded loss class...
متن کاملLearning theory: stability is sufficient for generalization and necessary and sufficient for consistency of empirical risk minimization
Sayan Mukherjee a,b, Partha Niyogi c, Tomaso Poggio a, and Ryan Rifkin a,d a Center for Biological and Computational Learning, Artificial Intelligence Laboratory, and McGovern Institute, USA E-mail: [email protected]; [email protected] b MIT/Whitehead Institute, Center for Genome Research, Massachusetts Institute of Technology, Cambridge, MA 02139, USA E-mail: [email protected] c Department of Computer Scien...
متن کاملStability Results in Learning Theory
The problem of proving generalization bounds for the performance of learning algorithms can be formulated as a problem of bounding the bias and variance of estimators of the expected error. We show how various stability assumptions can be employed for this purpose. We provide a necessary and sufficient stability condition for bounding the bias and variance for the Empirical Risk Minimization al...
متن کامل